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We consider a mixture of fermionic and bosonic atoms nearby interspecies Feshbach resonances, 
which have been observed recently in 6 Li- 23 Na mixture by MIT group, and in 40 K- 87 Rb mixture 
by JILA group. We point out that the fermion-boson bound state, namely the heteronuclear 
molecules, will coexist with the fermionic atoms in a wide parameter region, and the attraction 
between fermionic atoms and molecules will lead to the formation of atom-molecule pairs. The 
pairing structure is studied in detail, and, in particular, we highlight the possible realization of the 
Fulde-Ferrel-Larkin-Ovchinnikov state in this system. 



Introduction. In the latest few years ultracold atomic 
gases nearby Feshbach resonance have attracted consid- 
erable attention, since Feshbach resonance can be uti- 
lized to achieve strongly interacting quantum gases ex- 
hibiting universal properties, to produce Bose-Einstein 
condensation of molecules from atomic gases [li, and to 
study the physics of BEC-BCS crossover^ y|. While 
most present experimental and theoretical works focus on 
the resonance between atoms of the same species, inter- 
species Feshbach resonances were also observed recently 
in collisions between fermionic 6 Li and bosonic 23 Na by 
MIT group and in collisions between fermionic 40 K 
and bosonic 87 Rb by JILA group 5]. These interspecies 
Feshbach resonances arise from the fermion-boson bound 
states, which are called heteronuclear molecules. In this 
Rapid Communication, we will show that the fermionic 
atoms and molecules will coexist in a wide parameter re- 
gion, and they will form atom-molecule pairs and exhibit 
superfluidity, provided that the life-time of molecules can 
be long enough to reach equilibrium. Particularly, the 
system is a good candidate for realization of the Fuldc- 
Ferrell-Larkin-Ovichinnikov (FFLO) state, owing to the 
mass difference between atom and molecule, as well as 
the tunable molecular binding energy. 

In the study of superconductor with mismatched Fermi 
surfaces of different spins, FFLO state was proposed by 
Fulde and FerrellQ, and by Larkin and Ovchinnikov0 
independently. The distinct features of this state include 
(i) the center-of-mass momentum of each pair is nonzero, 
and thus the order parameter has a periodic space modu- 
lation; (ii) there are unpaired normal fermions coexisting 
with paired supcrfluid fermions. Stimulated by these ex- 
otic features, experimental condensed matter physicists 
have paid a lot of efforts on searching this state in vari- 
ous materials in the past decades, and until recently some 
evidences of discovering this state in CeCoIns have been 
reported 8]. This state has also been studied in the dis- 
cussion of color superconductivity and pulsars. These 
progresses have been systematically summarized in a lat- 
est review article Ref. Q . Recently, the realization of the 
FFLO state in the mixture of two-component fermions 
has been suggested [lJJ, ^J, where the Fermi surface 
mismatch is controlled by unequal population of differ- 
ent components, and the experimental signatures of the 
FFLO state in cold atoms experiments have also been 



discussed [TT| . Besides, there are also some other pro- 
poses for the pairing problem with mismatched Fermi 
surfaces, such as the breached-pair state[l2j and pairing 
with deformed Fermi surface 13] . Particularity the en- 
ergetic and dynamic stability of the breached-pair state 
have been discussed a lot in some recent literatures 

Setting up the model. We consider a uniform mixture 
of bosonic atoms (b.a.) and fermionic atoms (f.a.). Away 
from the resonance, the properties of such a mixture has 
been extensively discussed before|15|. By introducing the 
operator $ for the b.a., and the operator * a for the f.a., 
this mixture can be described by 
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where go is the strength of the self-interaction between 
b.a, gi is the strength of interaction between b.a. and f.a., 
and mi and mi are the masses of f.a. and b.a. respec- 
tively. Nearby the resonance, the process that converts 
two atoms into a heteronuclear molecule (h.m.) should be 
incorporated. In the spirit of the two-channel model [T^|. 
we introduce an independent fermionic operator \l/ m for 
this molecule, and describe the resonance process as 



Ham = a 
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And finally the molecules are described by 
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Here the mass m m of h.m., which equals mi + TO2, will 
be much larger than mi provided that m.2 is times larger 
than mi. g% is the interaction strength between h.m. 
and b.a., and the molecular binding energy /zo can be 
tuned by an external magnetic field. Therefore the total 
Hamiltonian is given by H = Ha + Hm + Ham- Since 
we are mostly interested in the behavior of the fermions 
in a bosonic background, we consider the situation that 
the density of bosons no is much larger than the density 
of fermions n, and therefore we can neglect the feedback 
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FIG. 1: (color online) The mean-field N — Eh phase diagram. 
Below the red solid line is the pure fermionic molecules phase; 
below the black dash line is the pure fermionic atoms phase, 
and in the region between the solid line and dash line atoms 
and molecules coexist. In the left-hand side of the dotted line, 
the Fermi momentum of atoms fcF a is smaller than the Fermi 
momentum of molecules fcFm- The two insets schematically 
show the dispersion relations e± (k) , with Eh < for the upper 
one and Eh > for the lower one. The momentum k is in 
the unit of fco = L/(2tv), where L is the length of the system, 
and the energy is in the unit of h 2 k$ / (2mi). All the figures 
in this paper are plotted for 40 K- 87 Rb mixture. 



on bosons safely. The opposite situation where n is com- 
parable to n has been considered recently 01 , an d the 
mean-field solution of the equal mixing case has also been 
provided 0. 

The mean-filed solution. At nearly zero temperature, 
the bosonic atoms are condensed. Adopting the standard 
Bogoliubov approximation, we expand the operator $ 
as 



- J2k e $k- Neglecting some constants, the 
Hamiltonian at the mean-field level turns out to be Ho = 
E k ea(k)*l k * Qk + e m (k)^ k * mk + aV^(*l k *mk + 
^Lk*ak), where e a (k) denotes h 2 k 2 /(2m 1 ), and e m (k) 
denotes h 2 k 2 /(2m m ) + Ef>. Here the binding energy /j, 
is normalized to by the mean-field interactions, i.e. 
E>b = f-o — 9i n o + .92^0- This quadratic Hamiltonian can 
be easily solved, which yields 
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and they are shown in the insets of Fig. ^ Owing to the 
large mass difference, |e a (k) — e m (k)| is usually very large 
unless k is very close to k c where e tt (k) crosses e m (k), the 
dispersion relations e± (k) thus can be well approximated 
by e Q ( m )(k) when \k— k c \ > 2a^/n ~m 1 m m / '(m 2 h 2 k c ), as 
can be seen from the Fig. ^ For a narrow resonance, this 
approximation is valid in the momentum region where 
the atom-molecule pairs are formed. 

It is noticed that the total number of fermionic atoms, 
including those in the scattering states and in the bound 
state, is conserved during the resonance process, i.e. 



FIG. 2: Upper: A diagrammatical illustration of the conven- 
tional scattering processes (a-b) and the Feshbach resonance 
process(c). Lower: The effective interaction between fermions 
owing to exchange of density fluctuations. The straight solid 
lines denote the propagate of Fermions, the twist solid lines 
denote the propagate of uncondensed Bosons, while the dot- 
ted lines denote the condensed Bosons. 



^i^ a + ^m^m = N, and these fermions fill the Fermi 
sphere with respect to e±(k) at zero temperature. Thus 
we obtain the mean-field phase diagram for different N 
and E\, shown in Fig. 2] from which one can see that 
in a wide range the atoms and molecules coexist in the 
system. This feature, contrasting with the Feshbach res- 
onance between the atoms of the same species, arises 
from the fact that both atoms and molecules in this sys- 
tem should obey the Pauli exclusive principle. Moreover, 
the Fermi surfaces difference Akp, which is defined as 
k Fm - k Fa , is given by ( v /2m m (^ - E h ) - y/2mifi)/H for 
fixed chemical potential fi, and increases as the decrease 
of E^. This is of crucial importance for the discussion of 
atom-molecule pairng below. 

Condensate fluctuation and the attractive interactions. 
We now incorporate the coupling between fermions 
and the condensate fluctuations illustrated in Fig. 
which includes (a) density-density interaction between 

f.a. and b.a., ffiVnoW^ ka *o I k 1 (*iLi-k a + *k 2 - kl ); (b) 
density-density interaction between h.m. and b.a., 



ft^m,k^m,k 1 ($l tI _ k2 + $k 2 -kj, and (c) the Fes- 
hbach resonance, a(& m k2 $k 2 -ki^a,ki + h.c). Consid- 
ering the second order processes that exchange bosonic 
fluctuations between fermions, an effective interaction 
between f.a. and h.m. will be induced |l9j|. which can 
be described by following effective Hamiltonian 

^ = E Z TW (« 2 *L,k 1+ k*i,k 2 -k*a, kl * m ,k 2 

kl k 2 k 4ff«o + 2^7 v 

-2g ig2 n ¥ mM _^ aM+ ^ aMl i> mM ^ .(5) 

Given the typical values of g\ and g 2 , when the density 
of bosonic atoms no is as large as 10 16 cm -3 for the width 
of resonance around I G, or when the width of the res- 
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onance is as narrow as lOmG for the density of bosonic 
atoms around 10 14 cm -3 , 2gig 2 n will be larger than a 2 
and the interaction between atoms and molecules will be 
attractive. This attraction will lead to the formation of 
atom-molecule pairs, and consequently the superfluid or- 
der (tf„# m ). 

Before proceeding, we should remark that not only the 
attraction between f.a. and h.m., but also the attraction 
between f.a.( h.m.) themselves will be induced. Indeed 
these terms will result in an atom-atom (or molecule- 
molecule) pairing order Q2 k ^o,k^o,-k}- However, In 
this work we focus on the pairng structure and superfluid 
properties of atom-molecule paired state because the p- 
wave pairing between atoms (molecules) is usually weak. 

Atom-molecule pairing. The following discussion on 
the pairing problem is based on the effective Hamilto- 
nian for the low-energy fermionic degrees of freedom, 
that is, H c tf = Ha + H lnt . For the low-energy scatter- 
ing processes where the exchanged momentum k is much 
smaller than ^/Egn^ns/h, we can approximate H- lnt by 
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der parameter A q = V(^ t *I i. , 



k 2 , with V denot- 
es 2 )/(4gn ). Following the idea of FF 
j, we start with a generalized BCS or- 

k «a,k+q*m,-k+q>' whe ™ Q 

is introduced as a variational parameter characterizing 
the center-of-mass momentum of pairs. With the stan- 
dard BCS mean-field theory, we can obtain the excitation 
spectrums E±(k) as 
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where fi is the chemical potential. Contrary to the con- 
ventional BCS superconductor case, here E±(k) are not 
always positive for arbitrary A q , and the pairs will break 
where the quasi-particle excitation energy is negative. 
Therefore the wave function should be taken as 



m,— k+q 



w= n *u, n 

{k|i5 + (k)<0} {k|B_(k)<0} 

n («k+«k< k+q <-k +q )io>. (7) 

{k|£±(k)>0} 

The order parameter A qi as well as the depairing regions, 
should be determined self-consistently from following gap 
equation 
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where Q(x) is the step function, with 0(x) = 1 for x < 
and 0(x) = for x > 0. 

In the Fig[3f a) we plot the right-hand side of the gap 
equation Eq.© as a function of A q with different val- 
ues of q, from which we can know all the possible solu- 
tions to the gap equation. We find that the equation 
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FIG. 3: (color online) (a) The right-hand side (R. H. S.) of the 
gap equation Eq.JHJ as a function of A with different values of 
q. q = for the dash-dotted line, q — 0.175fcF a for the dotted 
line, q = 0.255&Fa for the solid line and q = 0.35fcFa for the 
dash line, (b) The solutions of the gap equation Eq.JHJ, Aq, 
as a function of q. Both figures are plotted for the chemical 
potential fi fixed at Efz, A&f = 0.41/cFa and V = 8 x 10 _8 £ , Fa. 
The unit of q is fci?a, and the unit of A, fi and V is taken as 
Efh = fi 2 &Fa/(2mi) with fi 2 /(2mi) setting as unity. 
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FIG. 4: (color online) A illustration of the distribution of 
atoms(a) and the distribution of molecules(b) in the FFLO 
state. Here A/cf = 0.41A)Fa, q = 0.255fcFa, and V = 8 x 
10- 8 £ Fa . 



for Ao has two solutions. The one of large A is the 
BCS solution, where Cooper pairs are formed around 
fcp = [(fcp a + and the excitation spectrums 

are all gapped; and another of small A corresponds to 
the breached-pair state. The energy of the breached- 
pair state has been extensively studied before for similar 
modelO and will not be repeated here. The curves with 
small q, such as q = 0.175/cFa, are similar to the curve 
with q = 0. However, as q increases the gap equation has 
only one solution at small A, which corresponds to the 
FFLO states. The values of A q are obtained numerically 
for different q, and shown in Fig^b). The small A so- 
lutions, including the breached-pair state and the FFLO 
states, are all characterized by the existence of the de- 
paring region, which can be found from Fig^l where the 
particle populations of a typical FFLO state are shown. 
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FIG. 5: (color online) (a) The momentum per particle Q/N 
of the FFLO states for different values of q. A&f is fixed at 
0.4ffcF a . (b) The free energy per particle F = E/N — fi as a 
function of AA:f for BCS state (dotted line) and the FFLO 
states with q = 0.255&Fa (solid line) and q = 0.30fcFa(dash 
line). We change A/cf by tuning E\,. Both figures are plotted 
for V = 8 x fCP 8 -EFa- The units of q, Q and A&f is &Fa an d 
the unit of F is -Efel- 



In the FFLO state, the center of Fermi sphere moves 
to q. The atoms pair with molecules at the small mo- 
mentum side of the Fermi sphere, while the pairs at the 
high-momentum side will break. In this way, the FFLO 
states find a proper balance between lowering the interac- 
tion energy and not costing too much kinetic energy, and 
the nonzero momentum carried by the pairs can be can- 
celled out by the depaired normal molecules, producing 



an energetic stable FFLO state with the total momen- 
tum Q vanishing. We calculate the total momentum of 
the FFLO states for different values of q and find sta- 
ble state exists for large Afcp, such as Afcp = 0.41fcp a as 
shown in Fig. |S[a), and in Fig. EJb) we a l so compare the 
energy of two FFLO states with the BCS state, and find 
the FFLO state will dominate over the BCS state with 
the increase of Afcp. 

Remark. Although we have demonstrated the exis- 
tence of non-BCS type pairing state in certain region, 
the phase diagram of this system in the full parameter 
region is much richer. With the increase of E^, the sys- 
tem will enter a pure atomic phase, the atom-molecule 
pairing will be replaced by the p-wave pairing between 
fermionic atoms. Provided that the electric moments of 
these heteronuclear molecules are polarized by an exter- 
nal electric field, the direct dipole-dipole interactions be- 
tween molecules can not be ne glec ted, and it will lead to 
the pairing between molecules |2fJ. The competition be- 
tween these different supcrfluid orders will manifest itself 
in quantum phase transitions, and more fruitful physics 
is expected from the interplay of these phases, which will 
be a subject for further inverstigations. 
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